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Abstract - Probabilistic engineering design enhances safety 
and reduces costs by incorporating risk assessment directly into 
the design process. In this paper, we assess the format of the 
quantitative metrics for the vehicle which will replace the Space 
Shuttle, the Ares I rocket. Specifically, we address the metrics 
for in-flight measurement error in the vector position of the 
motor nozzle, dictated by limits on guidance, navigation, and 
control systems. Analyses include the propagation of error 
from measured to derived parameters, the time-series of dwell 
points for the duty cycle during static tests, and commanded 
versus achieved yaw angle during tests. Based on these 
analyses, we recommend a probabilistic template for specifying 
the maximum error in angular displacement and radial offset 
for the nozzle-position vector. Criteria for evaluating 
individual tests and risky decisions also are developed. 

I. Introduction 

T HE National Aeronautics and Space Administration 
(NASA) plans to retire the Space Shuttle in 2010. Under 
NASA’s Constellation Program, the Orion spacecraft and 
Ares series of rockets are set to replace the shuttle as the 
principal method for transporting astronauts and cargo into 
space. NASA’s Langley Research Center in Hampton, 
Virginia, is responsible for developing the static testing 
experiments required to evaluate the capabilities of Ares I 
rocket motors. The University of Virginia was tasked with 
specifying Ares I performance for: 1) thrust- vector angular 
misalignment; and, 2) thrust-vector radial offset during 
firing. Limits are required on each of these variables in 
order to comply with the limits on guidance, navigation, and 
control systems during flight. 

Specification is based on principles of probabilistic 
engineering design applied to statistical analysis of the Ares 
I static tests in order to verify performance. Traditionally, 
engineers design to input variables which are deterministic - 
they are set at either worst-case or nominal values. 
Calculating reliability based on worst-case values results in 
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over-design and unnecessarily large development and 
production costs. Alternately, adopting nominal scenarios 
requires that a factor of safety be applied to the resulting 
nominal design. Prudent safety factors also result in over- 
design and attendant costs. Without a rigorous treatment of 
the uncertainties in the input variables, there is no way to 
measure the potential over-design penalty in either case. 

By contrast, probabilistic engineering design addresses 
uncertainties explicitly [1], [2], [3]. Probability distributions 
are estimated for uncertain input variables. These estimates 
are propagated through the analysis, resulting in distributions 
for the output variables. Because this approach directly 
influences reliability, instead of designing to the worst case 
or safety-adjusted nominal, a probabilistic risk assessment is 
built into the design process. The probabilistic methodology 
provides a greatly improved understanding of reliability 
bounds which, in turn, can both enhance safety and reduce 
costs. 

The work reported here focuses on the analysis of 
previous static tests, in order to determine how these and 
future tests interface with probabilistic performance metrics 
of the Ares I design. Since test data are costly and scarce, 
sophisticated inference is required to determine whether 
performance has been met under acceptable levels of risk. 
Materials describing the static tests were reviewed and errors 
were detected in derivation of the static model [4]. It was 
determined that these errors were not incorporated into the 
static model and that the model appears to be sound under 
the reported assumptions. This report develops propagation- 
of-error, dwell-point, and input-output analyses, continues 
with a template for writing unambiguous and testable 
probabilistic metrics, and closes with a discussion of 
decisions and risk equilibrium. 



Fig. 1. Fire and smoke billow from a space shuttle reusable solid rocket 
motor static test at a Utah test facility. [5] 



II. Propagation-of-Error Analysis 


With respect to the functional relationship between the 
derived parameter and its measured constituents, derived 
parameter uncertainty is the aggregate of the error in each 
measured parameter. Since functional relationships are often 
based on derived parameters, propagated error directly 
influences whether a design will be accepted or rejected, 
correctly or incorrectly. 

Consider a motor nozzle vectored during static testing. 
There are two variables which determine angular 
displacement, ANGDISP (see Fig. 2): THETAY and SDELY. 
THETAY , the dynamic thrust vector, is the angle between the 
test-bay centerline and the motor thrust vector determined 
from longitudinal and lateral test stand thrust measurements 
(F x and F Y , respectively), given by: 


THETAY = arctan 
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SDELY , the nozzle vector angle, is the angle between the 
test-bay vector and the nozzle centerline calculated using 
nozzle extensometer data [6]. ANGDISP is defined as the 
difference between these two parameters: 

ANGDISP = THETAY - SDELY . 



Fig. 2. Line diagram depicting parameters of interest ( ANGDISP , THETAY , 
and SDELY) for the propagation-of-error analysis. [6] 


Deterministic target performance for ANGDISP is given by: 
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where t is time measured from the beginning of the test 
firing. We wanted to know the proportion of this ANGDISP 


metric potentially consumed by measurement errors. In the 
Appendix, we determined the functional form for the error in 
ANGDISP , as propagated through errors in THETAY and 
SDELY: 
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From this inequality, we see that ( ceteris paribus ) as 
longitudinal thrust, F x , increases, the error in ANGDISP is 
driven to, at most, the error in SDELY (since the uncertainty 
in SDELY is unknown). This result is likely in static testing 
scenarios as the magnitude of longitudinal thrust is much 
greater than that of lateral thrust for t > t h To determine the 
maximum error in ANGDISP under unlikely situations, we 
evaluated the inequality with maximum and minimum values 
for yaw and thrust, respectively, from previous static tests for 
t > tj. Thus, we set yaw to the maximum yaw angle from 
static test 14 and thrust to nominal minus 25 percent; o FY and 
o FX are set to known values. Thus, the error of ANGDISP is 
given by: 


G ANGDISP ~ XXX + (7 sdely j 


where XXX is small relative to the uncertainty in SDELY. 

Radial offset at the throat plane ( TRO ) is defined as the 
offset distance between the nozzle centerline and the 
dynamic thrust vector centerline determined at the throat 
plane and is approximated by: 


TRO~ 
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where FSF is forward side force, L Y is distance between the 
forward and aft test stands, and Y 0 o is nozzle centerline offset 
in the y-axis (as determined from the extensometers - this 
parameter is a function of SDELY). Deterministic target 
performance for TRO is given by: 

\TRO\ < X in , 


for all t. In the Appendix, we determined the functional form 
for the error in TRO , as propagated through the errors across 
its constituent parameters: 
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Once again, we see that ( ceteris paribus) as longitudinal 


thrust, F x , increases, the error in TRO is driven to, at most, 
the error in Y 00 (since the uncertainty in Y 0 o is unknown). 
Thus, uncertainties in ANGDISP and TRO are almost entirely 
dependent on uncertainties in SDELY or a function thereof 
( Y 0 o ), even in an extreme-case yaw/thrust scenario. Thus, if 
the uncertainty in SDELY is extremely large, improvements 
to the extensometers which improve their precision may be a 
viable option to prevent rejection of a suitable design. 
Otherwise, improvements to the nozzle positioning hardware 
may be more crucial to ensuring that ANGDISP and TRO are 
within specification during the static tests. 

III. Dwell-Point Analysis 

The duty cycle for a static test is designed to characterize 
a motor’s performance with limited resources. The duty 
cycle serves as a representation for various motor 
movements within flight (see Fig. 3). Within a given static 
test, the yaw angle oscillates between X±Y degrees for t = t 1 
to t 2 . Throughout this period, there are seven similar dwell 
points which occur at each crest and trough of the oscillatory 
wave. Since the vectored motor is never completely 
motionless, a dwell point is thus defined as a small time 
interval within the oscillatory period during which yaw angle 
does not deviate more than X degrees. The primary purposes 
of this dwell point analysis are to: 1) compare the within-test 
and across-test variation; and, 2) determine if the presence of 
an oscillatory pattern within a duty cycle provides 
information that cannot be gathered by other designs. 



Fig. 3. Current reusable solid rocket motor duty cycle for Ares I static 
testing. Dwell points occur in sawtooth pattern towards middle of graph. 


First, we performed an ANOVA to compare within-test 
ANGDISP values at each of the dwell points. The ANOVA 
showed that the ANGDISP values were statistically different 
across dwell points within each test (see Fig. 4). Further box 
plot analysis revealed that ANGDISP values were 
significantly different across runs at like dwell points (see 
Fig. 5). Yaw angles and ANGDISP values suggest that static 
test 1 1 was most unlike the other three tests. After the data 
from this test were removed from the set, additional ANOVA 
and box plots were generated. The data from the remaining 
runs show that ANGDISP still differs significantly across 


tests, even with the removal of test 11. 

In order to characterize ANGDISP values within and 
across runs, we plotted the yaw angles of dwell points from 
static tests 11-14 and discovered that: 1) the yaw angles at 
dwell points appear to increase within runs; and, 2) for a 
given dwell point, yaw angles are different across runs. 

The differences in yaw angles across tests are statistically 
significant, but their practical significance is suspect. Prior 
to this work, NASA knew that yaw angle is not constant 
within and across tests, and understood that nominal yaw 
angle at the dwell points according to the duty cycle is a 
nominal target. Additionally, the parabolic trend seen for 
yaw angle within tests was classified by NASA as a known 
phenomenon. 


ANGDISP at Dwell Points (Test 14) 



Dwell Points 


Fig. 4. Box plot of ANGDISP at dwell points for static test 14. These data 
exhibit within-test variation (an increase in ANGDISP over t). 


Both the crest and trough analyses show that yaw angles in 
static test 11 are significantly lower than in the other three 
tests. Additionally, ANGDISP values are not equal to one 
another across runs at a statistically significant level, with 
those in static test 11 being higher than in the other three 
tests. The correlation between ANGDISP and yaw angle is 
explored in the following section. 


Boxpiotof ANGDISP at Dwell 1 



Test 


Fig. 5. Box plot of ANGDISP at dwell point 1 across tests. ANGDISP in 
test 1 1 are statistically (but may not be practically) significantly different 
than ANGDISP in the other tests. 


From this analysis, we conclude that: 1) yaw angles at 
dwell points increase across time within tests and differ 
across tests, a phenomenon observed by NASA; 2) 
ANGDISP is not correlated with yaw angle (at this point in 
the analysis); and, 3) within each test, yaw angle increases 
along t for oscillatory dwell point data. Furthermore, at 
Dwell 1, static tests 12-14 have ANGDISP well within the 
error range calculated in the first section. Moreover, 
ANGDISP in static test 11 is within the target unless the 
uncertainty from the error propagation formulae is 
exceedingly large (more than three times the estimated 
value). The difference among the tests suggests that there is 
across-test (rocket-to-rocket) variation in these data, which 
could be caused by the control systems, rocket hardware, 
fuel mixture, or a host of other factors. 

IV. Input-Output Analysis 

Each trial’s yaw pitch angle distributions have the same 
approximate multimodal shapes, but there are discrepancies 
in the frequencies of commanded angles (consider dwell- 
point analysis in Section III). These discrepancies affect the 
theoretical output distributions of ANGDISP and TRO , which 
make across-trial analysis and inference difficult. That is, 
both test-to-test variation from exogenous variables and 
variation from command variables are present in TRO and 
ANGDISP. 

In Fig. 6, the four scatterplots of angular misalignment 
versus yaw exhibit positive slopes. That is, as yaw angle 
increases, angular misalignment increases, as well. Simple 
linear regression models were fit to these data to confirm this 
observation. In each trial, the slope of the regression model 
was found to be significant and positive. 


angular misalignment and radial offset increase, as well. 

In order to identify when during each firing this occurs, 
the residuals from the linear regression models were plotted 
against time. This revealed that large residuals come 
towards the end of the firing cycle for both ANGDISP and 
TRO , and also at the beginning of the firing cycle for TRO. 
Since pitch is radically manipulated at the end of the duty 
cycle, ANGDISP and TRO were each plotted against pitch 
for the four test firings. The most extreme variance occurs 
when pitch is approximately zero, which suggests another 
factor influences ANGDISP and TRO variability at the end of 
the duty cycle, which we hypothesize is change in thrust. 
This is confirmed by the contour plot in Fig. 8. 


The contour plot in Fig. 8 also brings to attention a bias in 
ANGDISP caused by motor nozzle orientation, as there is no 
symmetry about the Yaw = c axis. 



Fig. 7. Scatterplots of TRO vs. Yaw. 


Scatterplots of ANGDISP vs. Yaw 
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Fig. 6. Scatterplots of ANGDISP vs. Yaw. Notice how the slope of the 
ANGDISP vs. Yaw relationship for static test 1 lis steeper than those of 
static tests 12 through 14. 

In Fig. 7, the four scatterplots of radial offset vs. yaw 
appear to exhibit a negative slope. That is, as yaw angle 
increases, radial offset decreases. Simple linear regression 
models were fit to these data to confirm this observation. In 
each trial, the slope of the regression model was found to be 
significant and negative. As yaw increases, variance in 


Contour Plot of | ANGDISP | for Test 11 



Pitch (in degrees) 


Fig. 8. Contour plot of the absolute value of ANGDISP vs. Yaw vs. Pitch 
for test 11. \ANGDISP\ is greatest when Yaw is large and pitch is small. 
This suggests ANGDISP is affected by the sawtooth pattern in the duty 
cycle. 

V. Writing Probabilistic Metrics 

Typical performance metrics for ANGDISP and TRO 
could be written as follows [6] : 

ANGDISP: Misalignment of the dynamic thrust vector with 


respect to the nozzle centerline , shall deviate less than or 
equal to +X.XX degrees for T-t 7 to T=t 2 and +Y.YY degrees 
for T>t 2 ... 

TRO: The radial offset between the dynamic thrust vector 
and nozzle centerline at the movable nozzle throat plane 
shall be less than or equal to X.XX inches... 


failures from continuing operations, 0 is the probability of 
Type II error (the probability of incorrectly accepting an 
unsuitable design or “consumer’s risk”), and n is the 
expected probability of the unsuitable design causing failures 
of cost C F . 

The ideal scenario, in which the decision-making process 
is risk-neutral, occurs when: 


These metrics could be refined from a deterministic 
paradigm to address the probabilistic nature of acceptance 
sampling. As these are written, any datum recorded in a test 
firing which exceeds the specifications causes the run to be a 
failure. We refined the metrics to address the probabilities 
of both Type I and Type II error. Reliability, p, is the 
minimum proportion of data within specification and (p is a 
parameter of interest. The design goal is to obtain a 
proportion of (p above or below some threshold, (p MIN or 
Vmaxi which meets or exceeds the minimum reliability, which 
is related to Type I error: 

M^AmxMp = !-«]• 

Thus, design goals can be written in a probabilistic form. 
For example: 

P% ( reliability ) of all observations in a given test must be 
less than Y ( threshold value) with Z% of 0 risk ( determined 
by size of data pool ; Power =1-0). 


C D a = C F f5n. 

The decision-making process is risk-averse if: 

C D a>C F p7i, 

otherwise, it is excessively risk-seeking. Risk neutrality is 
the preferred long-run decision-making paradigm, as the 
expected value of the sum of incurred losses due to 
preventable delays and preventable failures is minimized. 
Accurate estimations of the cost parameters are essential to 
choosing a and 0, such that risk is neutralized. However, 
many costs are not explicit and do not scale with respect to 
magnitude. For example, what is the dollar value of an 
astronaut? Is it greater than that of an engineer or a civilian? 
Is the cost of delaying launch for 30 days the same as the 
cost of delaying launch for a day multiplied by 30? Biases in 
parameter estimation lead to unaccounted risk, which may 
lead to unintentional risk-seeking or risk-averse decision- 
making. 


For ANGDISP and TRO: 


Appendix 


ANGDISP: P% of observations of misalignment of the 
dynamic thrust vector with respect to the nozzle centerline , 
shall deviate less than or equal to ±X.XX for T-t, to T=t 2 
and +Y.YY degrees for T>t 2 with Z% of0 risk. 


A.l Propagation of error in ANGDISP: 

O ANGDISP — ® THETA Y + & SDELY 

where... 


TRO: P% of observations of radial offset between the 
dynamic thrust vector and nozzle centerline at the movable 
nozzle throat plane shall be less than or equal to X.XX 
inches with Z% of 0 risk. 

Reliability and desired power are parameters which are 
determined by project managers. Selection of these 
parameters involves careful risk and cost-benefit analyses. 

VI. Decisions and Risk Equilibrium 

In order to characterize the risk attitude of a mission- 
critical decision relating to performance, we developed the 
following expressions: 
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F y = Thrust x sin(90 - Yaw) , F x = Thrust x cos(90 - Yaw) 
assume... 


C D a and C F f3n ; , 

where C D is the expected cost of preventable delays from 
suspending operations, a is the probability of Type I error 
(the probability of incorrectly rejecting a suitable design or 
“producer’s risk”), C F is the expected cost of preventable 


p Fy Fx = 0 because load cells are rigid 
thus... 
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A.2 Propagation of error in TRO : 
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